Content-aware halftone image resizing

ABSTRACT

As provided herein, there are supplied teachings to systems and methods for resizing a halftone image using halftone tile parameters. One approach entails receiving into a digital imaging system, a digital halftone image and a desired resizing factor for the digital halftone image. Subsequently the system will define cells within the digital halftone image and determine from those cells, a number of halftone tile seams to suitable for manipulation. The orientation of these halftone tile seams is dictated by the received desired resizing factor. The energy of the number of halftone tile seams is determined according to an energy metric so as to provide indication of at least one low energy determined halftone tile seam. A resizing of the halftone image is then performing by manipulating at least one low energy halftone tile seam in the halftone image. The resized halftone image may then be printed on a printer.

CROSS-REFERENCE TO COPENDING APPLICATIONS

Attention is directed to copending applications filed concurrentlyherewith: entitled “Content-Aware Halftone Image Resizing usingIterative Determination of Energy Metrics”; U.S. application Ser. No.12/242,119, entitled “Content-Aware Resizing of Uniform Rosette ColorHalftone Images”; U.S. application Ser. No. 12/242,183, entitled“Content-Aware Uniform Rosette Color Halftone Image Resizing UsingIterative Determination of Energy Metrics”; U.S. application Ser. No.12/242,222, the disclosures found in these copending applications ishereby incorporated by reference in their entirety. The appropriatecomponents and processes of the above co-pending applications may beselected for the teaching and support of the present application inembodiments thereof.

BACKGROUND AND SUMMARY

The teachings herein are directed to a method and apparatus for resizinga halftone image using halftone tile parameters.

Demands imposed by today's digital media handling in regards to documentedit-ability, portability, and dynamic layout make simple solutions forimage resizing obsolete. Consider that a document can be rasterized andhalftoned for a particular print path. That document may be directed toa different printer, possibly years later when extracted from anarchive, with different paper size capabilities and may require imageediting, cropping and resizing of halftoned image content prior toprinting on the given print engine. Halftoned images may generally bere-purposed to print on different paper sizes and require layoutmodifications and resizing. Printed halftone images may be scanned in asetting such as at a digital copier, and a user may wish to modify theimage attributes such as size, aspect ratio, or image content.Conventional resizing methods that utilize spatially consistentinterpolation methods are unsuitable in this halftone image settingbecause interpolation methods introduce defects in halftone imagestructure and such spatially consistent interpolation can distort imagecontent.

To begin, consider the halftone image class of concern to the presentteachings herein. With the advent of inexpensive digital color printers,methods and systems of color digital halftoning have become increasinglyimportant in the reproduction of printed or displayed images possessingcontinuous color tones. It is well understood that most digital colorprinters operate in a binary mode, i.e., for each color separation, acorresponding color spot is either printed or not printed at a specifiedlocation or pixel. Digital halftoning controls the printing of colorspots, where the spatial averaging of the printed color spots by eithera human visual system or a viewing instrument, provides the illusion ofthe required continuous color tones.

The most common halftone technique is screening, which compares therequired continuous color tone level of each pixel for each colorseparation with one or more predetermined threshold levels. Thepredetermined threshold levels are typically defined for a rectangularcell that is tiled to fill the plane of an image, thereby forming ahalftone screen of threshold values. At a given pixel, if the requiredcolor tone level is darker than the given halftone threshold level, acolor spot is printed at that specified pixel. Otherwise the color spotis not printed. The output of the screening process is a binary patternof multiple small “dots,” which are regularly spaced as is determined bythe size, shape, and tiling of the halftone cell. In other words, thescreening output, as a two-dimensionally repeated pattern, possesses twofundamental spatial frequencies, which are completely defined by thegeometry of the halftone screen.

It is understood in the art that the distribution of printed pixelsdepends on the design of the halftone screen. For clustered-dot halftonescreens, all printed pixels formed using a single halftone celltypically group into one or more clusters. If a halftone cell onlygenerates a single cluster, it is referred to as a single-dot halftoneor single-dot halftone screen. Alternatively, halftone screens may bedual-dot, tri-dot, quad-dot, or the like.

While halftoning is often described in terms of halftone dots, it shouldbe appreciated that idealized halftone dots can possess a variety ofshapes that include rectangles, squares, lines, circles, ellipses, “plussigns,” X-shapes, pinwheels, and pincushions, and actual printed dotscan possess distortions and fragmentation of those idealized shapesintroduced by digitization and the physical printing process. Variousdigital halftone screens having different shapes and angles aredescribed in U.S. Pat. No. 4,149,194, “Variable Angle ElectronicHalftone Screening” to inventor Thomas M. Holladay, which is herebyincorporated by reference in its entirety.

A current practice for resizing an image is to perform some type ofresampling interpolation of the input image to generate an output imagewith the desired number of samples in each dimension. But, resampling ahalftone image with an interpolator such as nearest-neighbor, linear,quadratic, or cubic can result in defects within the halftone image. Oneparticularly problematic defect is the introduction of gray levels thatmust be re-halftoned prior to printing, where the re-halftoning stepcreates an interference pattern with the input halftone image structure.Another defect is the appearance of seams along columns or rows ofpixels, where the resampled halftone samples have a local disturbance inphase with respect to the halftone frequency. Yet another defectassociated with these resampling methods is that the aspect ratio ofimportant image content can become distorted. For example, use ofresampling to reduce the vertical dimension of an image can make peopleappear shorter and wider than they are in reality.

Another practice for resizing a halftone image is simply to crop thatimage to the desired size. However, cropping can delete desired imagecontent around the borders of the halftone image.

The diversity and versatility of display devices today imposes newdemands on digital media. For instance, designers must create differentalternatives for web-content and design different layouts for differentdisplay and rendering devices. These demands have lead to development ofincreasingly sophisticated image resizing tools for continuous tonedigital images. Avidan and Shamir, in “Seam Carving for Content-AwareImage Resizing” ACM Transactions on Graphics, Volume 26, Number 3,SIGGRAPH 2007, present a simple image operator called seam carving, thatsupports content-aware image resizing for both image reduction and imageexpansion. A seam is an optimal eight-connected path of pixels on asingle image from top to bottom, or left to right, where optimality isdefined by a low value of an image energy function. By repeatedlycarving out or inserting seams in one direction, Avidan and Shamir canchange the aspect ratio of an image. By applying these operators in bothdirections they can retarget the image to a new size. The selection andorder of seams protect the content of the image, as defined by theenergy function. Seam carving can also be used for image contentenhancement and object removal. The seam carving method of Avidan andShamir can support various visual saliency measures for defining theenergy of an image, and can also include user input to guide theprocess. By storing the order of seams in an image they createmulti-size images that are able to continuously change in real time tofit a given size.

The method of Avidan and Shamir cannot be readily applied to halftoneimages because selecting low energy seams will result in visuallyundesirable pathological seams that travel between halftone dots oralong chains of connected dots. Removing a low-energy seam that travelsbetween halftone dots would only increase the local darkness in theregion about the seam. Conversely, removing a low-energy seam thattraveled along connected halftone dots will decrease local darkness inthe region about the seam. In either case, the seams would appear asvisible streaks and would quite likely be objectionable. For example aninput halftone image where the pixels are at 600 dpi (dots per inch)resolution, and the halftone is at 141 cpi (cells per inch) at 45° whenresized by 10% in the horizontal direction by applying a low energy seamremoval method directly on the halftone image caused undesirable streaksto appear in the image.

One further option for resizing a halftone image is to apply adescreening technique to the halftone image to remove the halftone dotstructure and provide a continuous tone version of the image. Thecontinuous tone version of the image could be resized using the methodof Avidan and Shamir. After resizing, the image may be rehalftoned tofinally produce a resized binary image. However, a key problem with thatapproach is that any such descreening technique tends to blur finedetails within an image and the resulting image will have an excessively“soft” appearance. This softness problem will be particularly evidentwhen applied in binary printer image paths and copier image paths thatutilize a “copy dot” approach to reproduction. “Copy dot” refers todirect copying of a halftone image without descreening and rescreening.Resizing such a descreened image will induce a blurring that “copy dot”reproduction is intended to avoid.

As provided herein, there are supplied teachings to systems and methodsfor resizing a halftone image using halftone tile parameters. Oneapproach entails receiving into a digital imaging system, a digitalhalftone image and a desired resizing factor for the digital halftoneimage. Subsequently the system will define cells within the digitalhalftone image and determine from those cells, a number of halftone tileseams to suitable for manipulation. The orientation of these halftonetile seams is dictated by the received desired resizing factor. Theenergy of the number of halftone tile seams is determined according toan energy metric so as to provide indication of at least one low energydetermined halftone tile seam. A resizing of the halftone image is thenperforming by manipulating at least one low energy halftone tile seam inthe halftone image. The resized halftone image may then be printed on aprinter.

Also disclosed in embodiments herein is an image forming method forresizing a digital halftone image. The approach entails receiving into adigital imaging system, a digital halftone image and a desired resizingfactor for that digital halftone image. The system will then definecells within the digital halftone image and determine from those cells,a number of halftone tile seams to suitable for manipulation. Theorientation of these halftone tile seams is dictated by the receiveddesired resizing factor. The energy of these halftone tile seams is thendetermined according to an energy metric so as to provide indication ofa number of low energy determined halftone tile seams. The number of lowenergy seams identified is sufficient to achieve the desired resizingfactor. A resizing of the halftone image is then performing bymanipulating the low energy determined halftone tile seams within thehalftone image. The resized halftone image may then be printed on aprinter.

Further disclosed in embodiments herein is an image forming method forresizing a digital halftone image. The method entails receiving into adigital imaging system, a digital halftone image and a desired resizingfactor for that digital halftone image. The system will then definecells within the digital halftone image and determine from those cells,a number of halftone tile seams to suitable for manipulation. Theorientation of these halftone tile seams is dictated by the receiveddesired resizing factor. The digital halftone image is descreened toprovide an energy map. The energy of these halftone tile seams is thendetermined according to the energy map so as to provide indication of anumber of low energy determined halftone tile seams. The number of lowenergy seams identified is at least sufficient to achieve the desiredresizing factor for the digital halftone image. A resizing of thehalftone image is then performing by manipulating the number of lowenergy determined halftone tile seams within the halftone image toobtain a resized image. The resized halftone image may then be printedon a printer.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts an exemplary embodiment of a digital imaging systemsuitable for one or more aspects of the present invention.

FIG. 2 Schematic depicting an embodiment of the method taught herein.

FIG. 3 Schematic depicting relationship between pixels, halftone spatialvector representation, and parameters that relate to Holladayrepresentation.

FIG. 4 Schematic depicting relationship between pixels, a rotatedhalftone screen, a Holladay tile, and the parameters in the Holladayrepresentation.

FIG. 5 depicts an example of Holladay halftone tiles and a verticalhalftone tile seam connecting low energy tiles.

DETAILED DESCRIPTION

The teachings herein are directed to a method and apparatus for resizinga halftone image using halftone tile parameters. In particular, theteachings herein resize an image by manipulating halftone tiles or cellsalong low energy halftone tile seams. By using the tile parameters todefine the low energy halftone tile seams, a halftone image can beresized while avoiding the defects associated with applying knownresizing methods directly to halftone images.

In one embodiment of the teachings herein, a halftone image is receivedand a region of the digital halftone image is resized throughmanipulation (be it replicating or deleting) of low energy halftone tileseams within the halftone image. Halftone cell parameters are receivedor derived from analysis of the halftone image. A resizing factor orfactors are received and are used to determine if the manipulation isdirected to replicate low energy halftone tile seams or delete lowenergy halftone tile seams, and to determine the orientation of the seamto be manipulated (vertical or horizontal). The resizing factor and thecell parameters are used to determine what number of low energy halftonetile seams need to be removed or duplicated.

The energy of halftone tile seams within the halftone region isdetermined using an energy metric, such as the magnitude of thegradient. That energy metric may be derived on a pixel basis andspatially quantized to conform to halftone tiles, or it may be deriveddirectly from the tiles. The energy metric may also be a function ofdescreened values of pixels within the halftone image region. Thisenergy metric as determined for locations within the halftone imageregion is used to form an energy map.

The energy of the halftone tile seams is determined by using the energymap, where halftone tile seams are a connected path of cells that runeither vertically or horizontally through the image region depending onthe required resizing. The energy of a halftone tile seam is a functionof the integrated energy along the path of the halftone tile seam. Anumber of low energy halftone tile seams will be manipulated, where thenumber depends on the resizing factor and the halftone cell parameters.Resizing factors greater than 1 are defined to require low energyhalftone tile seam replication while resizing factors less than 1require low energy halftone tile seam deletion.

The low energy halftone tile seams are manipulated according to theirenergy ranking, where halftone tile seams possessing lowest orrelatively low energy are selected first for manipulation. A seam isconsidered a low energy seam if its energy is below a threshold, whichmay be a function of the energy level of the minimum energy seam. Lowenergy halftone tile seams can be manipulated either as a group, or themanipulation can be performed iteratively, where within the iterations,halftone tile seam energy is recomputed when one or more low halftonetile seams are removed or replicated, and the recomputed halftone tileseam energy is used for ranking and subsequent manipulation.

For a general understanding of the present invention, reference is madeto the drawings. In the drawings, like reference numerals have been usedthroughout to designate identical elements. In describing the presentinvention, the following term(s) have been used in the description.

The term “cell” indicates consistent unit area and shape of pixels orhalftone threshold values arrayed in manner that fills a region withoutforming gaps.

The terms “tile” and “Holladay tile” are used synonymously within thepresent disclosure, and indicate cells that are rectilinear in shape,constructed of vertical sides and horizontal sides. Due to thesecharacteristics they are convenient for use in horizontal or verticalresizing.

There are several equivalent representations for the periodic structureof a halftone image. For example, parameters that describe the tiling ofcells or parameters that describe the tiling of Holladay tiles bothfully describe the halftone structure for an image. Thus, “cellparameters” and “tile parameters” and related representations describedmore fully below possess equivalent information and transformations maybe made between the representations.

“Descreening” refers to methods to remove the halftone dot structure ofan image to produce a continuous tone version of the image.

“Eight-connectivity” commonly refers to pixels that touch neighboringpixels either on a side or a corner, where it is understood that anyflat edge of a tile or pixel is considered a side regardless oforientation. That is, if two pixels are adjacent, side-by-side orcorner-touching, they are said to be eight-connected. We extend thedefinition of eight-connectivity of pixels, to cells of pixels to referto a neighboring relationship between two cells where at least one pixelin the first cell is eight-connected to at least one pixel in the secondcell. An “eight-connected path” is an unbroken string of eight-connectedpixels or eight-connected cells.

A “seam” is an optimal eight-connected path of pixels on a given imagefrom top to bottom, or left to right, where optimality is defined by alow value of an image energy function. A “low energy seam” is a seampossessing an energy metric that is below a threshold level. Thethreshold level can be a function of the energy level of the minimumenergy seam, such as 20% greater than the energy level of the minimumenergy seam.

“Energy” refers to the variation in gray levels around a pixel orvariation in halftone cell values around a halftone cell. Neighboringpixels or neighboring cells that have similar values possess low energy,while neighboring pixels or neighboring cells that possess verydifferent values possess higher energy. An energy metric is a measure ofenergy, such as the sum of the magnitude of the gradient along aneight-connected path.

Within the field of halftoning, the term “screen” is sometimes used todenote an array of threshold values, and sometimes used to denote theperiodic structure of a halftone image. Within the present disclosure weare not concerned with a halftoning process, so use of the term “screen”herein will denote the periodic halftone structure unless the contextrefers to a screening process.

In FIG. 1, as depicted therein is one embodiment of a digital imagingsystem suitable for one or more aspects of the present invention. In thesystem 110, image source 120 is used to generate image data that issupplied to an image processing system 130 (including a digital frontend), and which produces output data for rendering by print engine 140.Image source 120 may include scanner 122, computer 124, network 126 orany similar or equivalent image input terminal. On the output endprinter engine 140 is preferably a xerographic engine however printengine 140 may include such equivalent print technology alternatives aswax, ink jet, etc. The teachings presented herein are directed towardaspects of image processor 130 depicted in FIG. 1. In particular, theintention of the teachings presented herein is to identify, and renderaccordingly, seams within a digital image. It will be appreciated bythose skilled in the art that the rendering of an image into a printableor displayable output format may be accomplished at any of a number oflocations, which herein is provided for in but one example only asoccurring within the image processing system 130 or within in the printengine 140.

Many applications can be envisioned within the system embodiment 110.For instance, consider that a document can be rasterized and halftonedby a computer 124 for a particular print path and printer 128. Thatdocument may be directed to a different printer 140, possibly yearslater when extracted from an archive 129, with different paper sizecapabilities and may require image editing, cropping and resizing ofhalftoned image content prior to printing on the given print engine.Halftoned images may acquired from a scanner 120 or retrieved from anarchive 129 as part of a document to be re-purposed. The document may bedirected to print on different paper sizes, and may be redesigned withdifferent layout requirements, thereby requiring resizing of its variouselements, such as the halftone image. Printed halftone images may bescanned using a scanner 122 in a setting such as at a digital copier,and a user may wish to modify the image attributes such as size, aspectratio, or image content prior to printing on a printer 140.

FIG. 2 provides a schematic for embodiments directed to the methodtaught herein. One embodiment of the method receives a halftone imageregion 310 and a resizing factor or factors. A factor indicates adesired magnification change for a particular orientation (e.g.,vertical or horizontal). The magnification factor corresponds to anumber of rows or columns of pixels to be deleted or replicated. Analternative to receiving a magnification factor is directly receivingthe number of rows or columns of pixels to be manipulated within theregion 320. Halftone tile parameters can optionally be input or they canbe estimated from the halftone image 330. The method applies adescreening operation 340 to the input halftone image to produce acontinuous-tone image. The descreening operation may use the halftonetile information so that the halftone texture may be well removed. Seamsare found within the contone image 350. The halftone tile seams aredetermined from connected halftone tiles or determined from connectedpixels and converted to connected halftone tiles. The low energy seamsof connected tiles (low energy halftone tile seams) are either removedor replicated 360 as needed in the input halftone image until 370 thedesired resizing is achieved and a resulting rescaled halftone image isoutput 380.

Halftone tile parameters may be known in some image processing settings.For instance, in a binary printer image path, the Digital Front End(DFE) 130 of the print engine 140 may possess one or more halftonethreshold arrays that are used to performed the halftoning operation.Knowledge of the halftone threshold arrays can provide the halftone tileinformation. If the halftone tile information is not known a priori, itcan be derived via a frequency analysis. For instance, digital imageprocessing analysis can be used to estimate the halftone cell parametersof a scanned halftone image in a digital copier setting. Varioustechniques have been applied to this estimation problem, such as findingthe maximum of an autocorrelation or Fourier transform, or performingwavelet analysis. Examples of methods for estimating halftone parametersare taught by Schweid in U.S. Pat. No. 6,734,991, Shiau and Lin in U.S.Pat. No. 7,239,430, and Lin and Calarco in U.S. Pat. No. 4,811,115. Thedisclosure found in these patents is hereby incorporated by reference inits entirety.

Key to the present invention is an understanding of halftoning tilingand halftone tile parameters. As will be shown, there are equivalentforms of halftone structure representation, such as frequency vectors,spatial vectors and Holladay tile parameters. Knowing one representationand its parameters allows determination of a form that is mostconvenient for the halftone tile seam manipulation. Due to therectilinear characteristics of Holladay tile parameters, they areconvenient for use in horizontal or vertical resizing. Due to theequivalence in representing halftone structure, within the presentdisclosure we often simply write “halftone tile parameters.”

Rationally angled periodic halftone screens can be represented inseveral equivalent forms. For instance, the periodic structure is oftendescribed by either two spatial vectors or two frequency vectorsspecifying the frequency and orientation of halftone periodicity. Forpresenting the teachings herein, a useful representation for halftoneperiodicity is the replication of rectangular cells with vertical andhorizontal sides. For rationally angled periodic screens, Holladay, inU.S. Pat. No. 4,185,304 has shown that there is such a rectangularhalftone cell representation that is equivalent to the spatial vectorand frequency vector representations. The disclosure found in U.S. Pat.No. 4,185,304 is hereby incorporated by reference in its entirety. Therepresentation of the equivalent rectangular cell requires only threeparameters in addition to the threshold values. We use the term“Holladay tile” to denote the tiles associated with halftone tilingutilizing rectangles having vertical and horizontal sides.

FIG. 3 teaches the relationship between halftone vector representation,which is often used to represent halftone cells aligned on a rationalangle, and rectangular tile representation. Let z₁ 410 be the spatialvector located in the first quadrant and let z₂ 420 be the frequencyvector located in the fourth quadrant. The halftone screen representedby the spatial vectors z₁ and z₂ can be thought of as the replicationsof parallelogram halftone cells with sides of length ∥z₁∥ and ∥z₂∥ atangles ∠z₁ 430 and ∠z₂ 440, where ∥z∥ is the norm of vector z and ∠z isits angle. Let z₁=(x_(i),y_(i)) be the rectangular coordinaterepresentation of the vector whose polar coordinate representation is(∥z_(i)∥,∠z_(i)), for i=1, 2. For a rationally angled halftone screen,x_(i) and y_(i) are integers. The area of each parallelogram isA=∥(x ₁ ,y ₁)×(x ₂ ,y ₂)∥=x ₂ y ₁ −x ₁ y ₂,  (1)where x is the vector cross-product operator.

Alternatively, frequency vectors f₁ and f₂ may be provided instead ofspatial vectors when specifying the halftone screen. Vector f_(i) isparallel to vector h_(i), which is the height of the parallelogramperpendicular to vector z_(i) for i=1, 2 as illustrated in FIG. 4. Themagnitude of vector f_(i) is equal to the reciprocal of the magnitude ofvector h_(i) for i=1, 2.

Let L be the shortest distance along the horizontal axis in which theparallelogram pattern recurs (points 3 450 and 4 460 in FIG. 4), and letL 470 be its corresponding vector. Since L 470 is obtained by followinga path along the sides of the parallelogram, it must be the result ofthe addition of an integer number of vectors z₁ 410 and z₂ 420.Mathematically,L=kz ₁ +jz ₂,  (2)where k and j are the smallest integer values that satisfy the equation.Decomposing Eq. (2) into its components, we getL=kx ₁ +jx ₂,  (3)0=ky ₁ +jy2.  (4)Let p be the greatest common divisor of y₁ and y₂. Thenp=−y ₁ /j=y ₂ /k≧1.  (5)Combining Eq. (5) with Eq. (3) and substituting into Eq. (1) gives areaA=Lp.  (6)Since A and p are known (either z₁ 410 and z₂ 420 or f₁ and f₂ areknown), L can be calculated.

A similar analysis in the vertical direction yields areaA=Kq,  (7)where K is the shortest vertical distance in which the parallelogrampattern recurs and q is the greatest common divisor of x₁ and x₂. SinceA and q are known, K can be calculated.

Equations (6) and (7) imply that an integer number of trapezoids witharea A fit inside a rectangle with dimensions K height and L width. Thisis an insight into the equivalence between halftone screenrepresentation using rationally-angled trapezoids and representationusing Holladay tiles.

Equation (6) implies that the area of one parallelogram is contained ina rectangular block of area A=Lp (see FIG. 4), which must repeat itselfa distance of p 510 units in the vertical direction but displaced in thehorizontal direction a certain amount D 520 to account for the slant ofthe original halftone screen so that it falls on the same position inthe angled grid defined by the screen (see points 1 and 3 in FIG. 4). Inorder to get from point 1 480 to point 3 450, a path defined by thevectors D 490 and p 495 may be followed; alternatively, a path obtainedby concatenating an integer number of vectors z₁ 410 and z₂ 420 can alsobe traced. In mathematical terms,D+p=m z ₁ +n z ₂,  (8)which has a unique solution for D 490, m and n if additional constraintssuch as m≧n and D≦L are imposed. Successive displacements by D 520 inthe horizontal direction and p 510 in the vertical direction give thelocation for the next rectangular cell of horizontal width L 530 witharea A. This means that, in effect, a rectangular tile and displacementcan represent a rationally-angled screen. The parameters that are neededto uniquely specify a halftone screen using rectangular tiles are L 530,p 510 and D 520.

It should be noted that the above analysis defined a rectangular tilingwith a row-to-row displacement in the horizontal direction. As will beappreciated by those skilled in the art it is straightforward to rotatethe analysis by 90 degrees to define rectangular tiling with acolumn-to-column displacement in the vertical direction.

Further, we must note that tiles that directly correspond to halftonefrequency vectors can be combined to form larger tiles that fill theimage plane. For instance, a halftone defined by a frequency vector of212 cells/inch at 45° constructed from pixels at 600 per inchcorresponds to a Holladay halftone tile that is 2 pixels in height, 4pixels in width, and offset in successive rows by 2 pixels. Those tilescan be groups in many ways to form larger tiles, sometimes referred toas supercells, or multi-centered cells, that cover the image. Oneexample is that two tiles in a row can be combined to form a dual-dottile that is 2 pixels by 8 pixels in dimension, and successive rows areoffset by 2 pixel. An alternative dual-dot tile can be constructed oftiles on two rows to be 4 pixels by 4 pixels in dimension with an offsetof 0 pixels. Those skilled in the art of halftone design can producemany alternative tilings by combining smaller tiles in variations. Withrespect to the teachings herein, resizing using the smallest possibletile is expected to yield the best image quality due to the fineness ofthe tile. On the other hand, a tile composed of multiple smaller tilesoffers the advantage of reduced computation because a larger number ofpixels are affected in each seam manipulation.

A descreening step is utilized in one embodiment of the teachings hereinto prevent the confounding of the “energy metric” that is used to findlow energy halftone tile seams in the image. Content-aware halftoneimage resizing finds low energy halftone tile seams that can be removedor replicated without drastically changing the appearance of the image,thus producing a graceful resizing of the original image. Descreeningproduces a continuous-tone (gray-scale) representation of the inputhalftone image region and can prevent the presence of halftone texturefrom interfering with the energy metric. Failure to prevent such textureinterference could lead to potentially erroneous decisions as to whichregions of the image have the least visible impact on the overallappearance. There are numerous descreening methods that can be utilizedby one skilled in the art to obtain a continuous-tone representation ofthe input halftone image. Examples of methods for descreening areprovided in Harrington in U.S. Pat. No. 6,864,994, Fan et al., in U.S.Pat. No. 6,839,152, and De Queiroz et al., in U.S. Pat. No. 5,799,112.The disclosure found in these patents is hereby incorporated byreference in its entirety for their teachings.

A seam is defined as a low energy eight-connected path that traversesthe image from top to bottom or from left to right. Eight-connectivitycommonly refers to pixels that touch neighboring pixels either on a sideor a corner, where it is understood that any flat edge of a tile orpixel is considered a side regardless of orientation. That is, if twopixels are adjacent, either as side-by-side or corner-touching, they aresaid to be eight-connected. We herein extend the definition ofeight-connectivity of pixels, to cells of pixels to refer to aneighboring relationship between two cells where at least one pixel inthe first cell is eight-connected to at least one pixel in the secondcell. An eight-connected path is defined as an unbroken string ofeight-connected pixels or eight-connected cells. A given seam as definedfor the purposes of the present disclosure can only have one type ofstrong connectivity, vertical or horizontal, where strong connectivityrefers to side-to-side touching, as opposed to corner touching, which isreferred to as weak connectivity. For instance, a seam that traverses animage region from top to bottom may not possess strong horizontalconnectivity, and vice versa. Within the teachings provided herein,where the distinction between a seam of pixels and a seam of halftonetiles is important, we refer to a seam constructed of halftone tiles asa “halftone tile seam.”

FIG. 5 illustrates a low energy halftone tile seam running verticallythrough a section of a digital image. The tiles are formed of 2×4 pixelblocks, where each successive row of blocks is offset by 1 pixel. In thefigure, the darkness of a tile represents the energy of that tile. Forexample, tile 610 is relatively dark and is illustrative of a highenergy tile, while tile 620 is relatively light and illustrative of alow energy tile. A vertical low energy halftone tile seam 630 is formedby connecting the centers of low energy halftone tiles along a path fromthe top of the image section to the bottom of the section. Removing orreplicating tiles along this path 630 would have the least noticeableimpact on the visual appearance of the image, and thus is desirable formanipulation for resizing.

The optimality of a seam depends on the desired application. In resizingimages according to the teachings herein, a desired image size isachieved by repeatedly removing or replicating low energy halftone tileseams that run in the appropriate direction. In this application, theoptimality of a halftone tile seam refers to the degree to which itwould not be noticed upon removal or replication. In this case, ahalftone tile seam is optimal if the pixels or cells that make up thehalftone tile seam are located in regions with little activity asmeasured by an energy metric. By activity, we mean variations in graylevel within a spatial region. A region with high gray level variationis said to have high activity and would possess a high value energymetric. In one embodiment of the teachings herein, the energy metric ateach pixel or cell is the magnitude of the local image gradient: thelarger the local gradient, the larger the energy at that particularpixel or cell. A discussion of the gradient is provided to aid inteaching the presently disclosed method, and is not intended to limitthe energy determination step to gradient-based metrics.

A monochromatic image can be seen as a function of two variables, f(x,y)whose value at (x,y)=(x₀,y₀) is the gray value of the pixel located inrow x₀ and column y₀ of the image. In one embodiment of the presentteachings herein, the energy metric at each pixel is determined as themagnitude of the local image gradient: the larger the local gradient,the larger the energy at that particular pixel. The gradient of atwo-variable function is defined as the two-dimensional vector

$\begin{matrix}{{{\bigtriangledown\;{f( {x,y} )}} = \lbrack {\frac{\partial f}{\partial x},\frac{\partial f}{\partial y}} \rbrack},} & (9)\end{matrix}$with magnitude

$\begin{matrix}{{{{\bigtriangledown\;{f( {x,y} )}}} = \sqrt{( \frac{\partial f}{\partial x} )^{2} + ( \frac{\partial f}{\partial y} )^{2}}},} & (10)\end{matrix}$where ∂f/∂x and ∂f/∂y are directional derivatives in the x and ydirection, respectively. Directional derivatives can be approximated byapplying spatial kernels that compute finite differences. Examples ofsuch spatial kernels are the Prewitt, which are described in“Fundamentals of Digital Image Processing” by A. Jain, Prentice Hall.1989:

$\begin{matrix}{\begin{bmatrix}{- 1} & 0 & 1 \\{- 1} & 0 & 1 \\{- 1} & 0 & 1\end{bmatrix},\begin{bmatrix}{- 1} & {- 1} & {- 1} \\0 & 0 & 0 \\1 & 1 & 1\end{bmatrix},} & (11)\end{matrix}$and the Sobel, which are described in “Digital Image Processing,” by R.Gonzalez and P. Wintz, Addison-Wesley Publishing Company, 2^(nd)Edition, 1987:

$\begin{matrix}{\begin{bmatrix}{- 1} & 0 & 1 \\{- 2} & 0 & 2 \\{- 1} & 0 & 1\end{bmatrix},{\begin{bmatrix}{- 1} & {- 2} & {- 1} \\0 & 0 & 0 \\1 & 2 & 1\end{bmatrix}.}} & (12)\end{matrix}$

The Laplacian, also described in the books by Jain, and Gonzalez andWintz, is another measure of local energy that is less computationallyexpensive since it only requires the application of one kernel.Mathematically, it is defined as

$\begin{matrix}{{{\bigtriangledown^{2}\;{f( {x,y} )}} = {\frac{\partial^{2}f}{\partial x^{2}} + \frac{\partial^{2}f}{\partial y^{2}}}},} & (13)\end{matrix}$and can be approximated with the kernel

$\begin{matrix}{\begin{bmatrix}0 & 1 & 0 \\1 & {- 4} & 1 \\0 & 1 & 0\end{bmatrix}.} & (14)\end{matrix}$However, due to the second-order derivative, the Laplacian operator ismore sensitive to noise than the gradient operators.

Note that other contone energy metrics may be used within the presentteachings, for example functions of the gradient, entropy, visualsaliency, eye-gaze movement, Harris corners, output of face detectors,and other specific feature detectors as discussed by Avidan and Shamirin “Seam Carving for Content-Aware Image Resizing” ACM Transactions onGraphics, Volume 26, Number 3, SIGGRAPH 2007.

In one embodiment of the teachings herein the method calculates theenergy metric at each pixel or cell of the continuous-tone image andthen searches for the seams with lowest energy content that traverse theimage from top to bottom for horizontal resizing and from left to rightfor vertical resizing. Several alternative embodiments of the low energyseam calculation are within the scope of the teachings herein. In oneembodiment, a low energy seam is determined directly from individualpixel values and spatially quantized to a halftone tile seam thatcontains the pixel-based seam. In another embodiment, the energy of thepixels within each cell is combined and the halftone tile seams aredetermined from the combined energy as found within the cell. In yetanother embodiment, the energy metric is computed from cell values,where integrated pixels values within each cell, produce the cell value.The energy metric in this embodiment is based on the difference betweenthe value of a cell and the values of its neighboring cells. This metriccan computed in a manner similar to using kernels as found in Eqs. (11),(12) and (14), but the weights in the kernel would be positioned to becompatible with the given cell tiling rather than the conventionalCartesian pixel tiling. Alternatively, direct application ofdifferencing equations, such as the Laplacian [Eq. (14)] can be appliedusing cell values and positions rather than the conventional pixel-basedmethod.

The energy metrics used for determining low energy seams discussed thusfar have been derived from a contone image representation of thereceived halftone image. It is within the scope of the teachings hereinto alternatively derive energy metrics and low energy halftone tileseams directly from the halftone image. For instance, a measure ofcell-to-cell difference is an energy metric. A useful cell-to-celldifference metric is a count of the number of pixels within a cell thatare of a different state than corresponding pixels within itsneighboring cells. Stated mathematically,

${E = {\sum\limits_{k}^{\;}{\sum\limits_{i,j}^{\;}{( {C( {i,j} )} ){{XOR}( {N_{k}( {i,j} )} )}}}}},$where E is the energy of the cell C, C(i, j) are the halftone pixelvalues of the cell at positions i, j, and N_(k)(i, j) are the halftonepixel values of the cell at corresponding positions i, j within the kthneighboring cell, and XOR is the Exclusive OR operation.

One method to determine an optimal seam from energy metrics, is toemploy an exhaustive search that calculates the energy of every possibleseam that originates at each pixel or cell of the first row (column) ofthe image. A more computationally efficient alternative approach is touse a dynamic programming method, because it can eliminate redundancywithin the exhaustive approach. This redundancy is present because seamsoriginating at neighboring pixels (or cells) may have multiple pixels(or cells) in common.

At each iteration, the dynamic programming approach traverses the energymap row by row to define vertical seams or column by column to definehorizontal seams and finds the cumulative minimum energy at each pixelor cell of each row (or column). At the end of the process fordetermining vertical seams, the smallest value on the last row indicatesthe end point of the optimal path that traverses the image from top tobottom. At the end of the process for determining horizontal seams, thesmallest value on the last column indicates the end point of the optimalpath that traverses the image from left to right. The remaining pixels(or cells) that belong to the seam are found by traversing the energymap in the opposite direction and picking the neighboring pixel (orcell) with the smallest cumulative energy in each subsequent adjacentrow for vertical seams or column for horizontal seams.

Once a low energy halftone tile seam is found, it will need to bereplicated or removed as depending on the resizing factor. To achieveacceptable visual appearance a halftone tile seam does not need topossess the absolute lowest energy to be selected for replication ordeletion. It is sufficient for the halftone tile seam to possessrelatively low energy as can be determined via an energy threshold. Forexample, halftone tile seams possessing energy below a threshold of 5%above the minimum seam energy as determined from the image can beselected for manipulation.

As previously noted, replicating or removing a single arbitrary pixel ina row or in a column of a halftone image will introduce resizingstreaks. In order to avoid the introduction of artifacts, the presentlytaught method manipulates low energy seams constructed of cells, ratherthan pixels. Resizing factors greater than 1 require low energy halftonetile seam replication while resizing factors less than 1 require lowenergy halftone tile seam deletion. A resizing factor that does notequal 1 in the vertical direction requires manipulation of horizontallow energy halftone tile seams, while a resizing factor that does notequal 1 in the horizontal direction requires manipulation of verticallow energy halftone tile seams. Resizing factors not equal to 1 may beprovided for each orientation, and low energy halftone tile seammanipulation could proceed in serial, one orientation after the other,or low energy halftone tile seam manipulation could alternate betweendirections until the adequate number of low energy halftone tile seamsis manipulated for each orientation.

Let the height of the halftone cell be p pixels and its width L pixels.Then at each iteration of low energy halftone tile seam manipulation fora resizing factor less than 1, the removal of a low energy halftone tileseam results in the removal of p pixels for vertical manipulation and Lfor horizontal manipulation. Similarly, at each iteration of low energyhalftone tile seam manipulation for a resizing factor greater than 1,the replication of a low energy halftone tile seam results in thereplication of p pixels for vertical manipulation and L for horizontalmanipulation. Thus, the image is decreased or increased in height orwidth by multiple pixels in each iteration. This implies that if it isdesired to decrease or increase the height or width of the image by mpixels, the process needs to be iterated [m/p] times for verticalmanipulation and [m/L] times for horizontal manipulation, where [●]denotes the rounding operator, which approximates to the nearestinteger. Alternatively, other rounding approaches, such as the nearestlower integer or nearest higher integer may be also be used. Thus, lowenergy halftone tile seam removal can achieve the desired resizing tothe accuracy of the dimensions of a tile since a tile is the unit ofmanipulation.

As we have noted, typically, multiple low energy halftone tile seamsmust be manipulated to achieve the desired resizing. There are severalalternatives within the scope of the present teaching for removingmultiple low energy halftone tile seams. The alternatives exist becauseremoving one or more seams can affect the energy metrics of other seams.The alternatives are important because some offer the advantage of alower burden of computation, while other alternatives may provide betterimage quality, and yet other alternatives are a compromise betweencomputational burden and optimal image quality.

A method for selecting multiple low energy seams that has a lowcomputational burden employs a one-time calculation of seam energy. Thatis, the energy for many halftone tile seams may be calculated one time,and the appropriate number of low energy halftone tile seams can beselected from halftone tile seams possessing lowest values using theenergy of the seams determined from the one-time calculation. Whenremoving multiple seams based on a one-time energy calculation, it mustbe noted that a halftone tile can be a member of one or more seams. Whenthis situation occurs, the seam is manipulated that possesses the lowestenergy. The other seams that contain this tile are then not manipulated.

In terms of perceived image quality, a very high quality resizingproduces an output that introduces the least abrupt density variationdue to seam removal. A method for selecting multiple low energy seamsthat has the potential for achieving the highest image qualityrecalculates seam energy after each manipulation of a low energyhalftone tile seam. This method ensures that the absolute lowest energyseam may be removed. Additionally, the issue of a tile belonging to twoor more seams having low energy does not occur.

Alternative methods for removing multiple seams are a compromise betweencomputational burden and optimal image quality. One of these compromisemethods recalculates seam energy after a fixed number of low energyhalftone tile seams have been manipulated. Another compromise approachto recalculating seam energy is to recalculate the energy when the nextlowest energy halftone tile seam that has not been removed issufficiently different in energy from low energy halftone tile seamspreviously removed. There are several ways that the difference can bedetermined. One method calculates the difference in energy with respectto the lowest energy seam found in the most recent seam energycalculation. Another method determines difference with respect to thehighest energy seam removed thus far. Other differences may also beuseful to consider, such as difference with respect to the mean energyof seam removed thus far. Differences on the order of, say 5% or greatmay merit an energy recalculation.

There are also several alternatives within the scope of the presentteaching for replicating multiple low energy halftone tile seams. Forexample, one low energy halftone tile seam possessing the lowest orrelatively low energy (for example, within 5% of the minimum energyseam) could be replicated until the resizing is achieved. For some imagetypes it could be desirable to choose low energy halftone tile seams forreplication that are spatially separated so that a noticeable band oflow energy is not inserted into the image. In this situation, somenumber of spatially separated low energy halftone tile seams could bereplicated to achieve the desired resizing. For example, it can bedesirable to replicate low energy halftone tile seams that are separatedby 3 or more halftone tiles.

The claims, as originally presented and as they may be amended,encompass variations, alternatives, modifications, improvements,equivalents, and substantial equivalents of the embodiments andteachings disclosed herein, including those that are presentlyunforeseen or unappreciated, and that, for example, may arise fromapplicants/patentees and others.

1. A method for resizing a digital halftone image, the methodcomprising: receiving into a digital imaging system, a digital halftoneimage and a desired resizing factor for the digital halftone image;defining cells within the digital halftone image; determining from thedefined cells, a number of halftone tile seams to manipulate, where theorientation of the number of halftone tile seams is dictated by thereceived desired resizing factor; determining an energy map of thedigital halftone image according to an energy metric; determining theenergy of the number of halftone tile seams within the energy map so asto provide indication of at least one low energy determined halftonetile seam; performing a resizing of the halftone image by manipulatingthe at least one low energy determined halftone tile seam, to obtain aresized halftone image; and, printing the resized halftone image on aprinter.
 2. The method of claim 1 where the methodology is only appliedto a sub-region of the digital halftone image.
 3. The method of claim 1where the at least one low energy determined halftone tile seampossesses energy below an energy threshold.
 4. The method of claim 3where the energy threshold is 5% greater than the lowest energydetermined for any of the number of halftone tile seams.
 5. The methodof claim 3 wherein the manipulating of the at least one low energydetermined halftone tile seam is a replication operation for increasingthe size of the digital halftone image.
 6. The method of claim 3 whereinthe manipulating of the at least one low energy determined halftone tileseam is a deletion operation for decreasing the size of the digitalhalftone image.
 7. The method of claim 5 wherein the manipulating of theat least one low energy determined halftone tile seam is a replicationoperation for a resizing factor greater than
 1. 8. The method of claim 6wherein the manipulating of the at least one low energy determinedhalftone tile seam is a deletion operation for a resizing factor lessthan
 1. 9. The method of claim 3 wherein a number of low energy halftonetile seams are manipulated such that this number results in the resizedhalftone image possessing the size dictated by the resizing factorwithin a multiple of the dimensions of a halftone tile.
 10. The methodof claim 3 wherein defining the halftone cells utilize halftone screenparameters and the parameters are derived from one of autocorrelation,frequency analysis, wavelet decomposition, or directly receiving thescreen parameters.
 11. The method of claim 10 wherein the halftonescreen parameters defining halftone cells are used to derive a Holladaytile representation of the cells comprising a width, height and offsetfor the cells, and the number of low energy halftone tile seams areconstructed of Holladay halftone tiles.
 12. The method of claim 10wherein the energy map is determined by calculating energy on apixel-by-pixel basis using descreened pixel values, and a halftone tileseam is determined by calculating a low energy pixel seam and quantizingthe pixel seam to a halftone tile seam.
 13. The method of claim 10wherein the energy map is calculated on a tile basis by integrating theenergy of descreened pixels belonging to each respective tile.
 14. Themethod of claim 10 wherein the energy map is determined on a tile basisby directly calculating tile-to-tile difference.
 15. The method of claim3 wherein the energy is measured according to a metric defined by one ofa function of, the gradient, entropy, visual saliency, eye-gazemovement, Harris corners or output of face detectors.
 16. The method ofclaim 3 wherein the energy of a halftone tile seam within the energy mapis determined by integrating the energy of eight-connected cells alongthe seam that traverses the image in the direction dictated by theresizing factor.
 17. The method of claim 7 wherein the resizing factoris greater than 1, thereby requiring a replication of low energy seams,and one low energy halftone tile seam possessing relatively low energyis replicated until the resizing is achieved, where low energy isdefined to be seam energy within 5% of the minimum energy seam.
 18. Themethod of claim 8 wherein the resizing factor is greater than 1, therebyrequiring a replication of low energy seams, and low energy halftonetile seams that are spatially separated by at least the width of threetiles are replicated until the resizing is achieved, where low energy isdefined to be seam energy within 5% of the minimum energy seam.
 19. Themethod of claim 3, wherein each tile used to form the number of halftonetile seams represents a multi-center halftone cell.
 20. The method ofclaim 3, wherein determining the energy of the number of halftone tileseams according to an energy metric is one of employing an exhaustivesearch and employing a dynamic programming method.
 21. A method forresizing a digital halftone image, the method comprising: receiving intoa digital imaging system, a digital halftone image and a desiredresizing factor for the digital halftone image; determining halftonescreen parameters for the digital halftone image; defining cells withinthe digital halftone image from the determined halftone screenparameters; determining from the defined cells and determined halftonecell parameters, a number of halftone tile seams to manipulate, wherethe orientation of the number of halftone tile seams is dictated by thereceived desired resizing factor; determining the energy of the numberof halftone tile seams according to an energy metric so as to provideindication of a number of low energy determined halftone tile seamssufficient to achieve the desired resizing factor for the digitalhalftone image; performing a resizing of the halftone image bymanipulating the number of low energy determined halftone tile seams, toobtain a resized halftone image; and, printing the resized halftoneimage on a printer.
 22. The method of claim 21 further comprising thestep of: descreening the digital halftone image to provide descreenedpixel values; and determining the energy metric from the descreenedpixel values.
 23. The method of claim 21 where the methodology is onlyapplied to a sub-region of the digital halftone image.
 24. The method ofclaim 21 where the number of low energy determined halftone tile seamsare halftone tile seams that possess energy below an energy threshold.25. The method of claim 24 where the energy threshold is 5% greater thanthe lowest energy determined for any of the number of halftone tileseams.
 26. The method of claim 24 wherein the manipulating of at leastone low energy determined halftone tile seam is a replication operationfor increasing the size of the digital halftone image.
 27. The method ofclaim 24 wherein the manipulating of at least one low energy determinedhalftone tile seam is a deletion operation for decreasing the size ofthe digital halftone image.
 28. The method of claim 24 wherein a numberof low energy halftone tile seams are manipulated such that this numberresults in the resized halftone image possessing the size dictated bythe resizing factor within a multiple of the dimensions of a halftonetile.
 29. The method of claim 24 wherein the halftone screen parametersdefining halftone cells are used to derive a Holladay tilerepresentation of the cells comprising a width, height and offset forthe cells, and the number of low energy halftone tile seams areconstructed of Holladay halftone tiles.
 30. The method of claim 22wherein determining the energy of the number of halftone tile seams isperformed by summing the energy of descreened pixels within eachrespective tile.
 31. The method of claim 24 wherein the energy of ahalftone tile seam is determined by integrating the energy ofeight-connected cells along the seam that traverses the image in thedirection dictated by the resizing factor.
 32. A method for resizing adigital halftone image, the method comprising: receiving into a digitalimaging system, a digital halftone image and a desired resizing factorfor the digital halftone image; defining cells within the digitalhalftone image; determining from the defined cells, a number of halftonetile seams to manipulate, where the orientation of the number ofhalftone tile seams is dictated by the received desired resizing factor;descreening the digital halftone image to provide descreened pixelvalues; determining an energy metric from the descreened pixel values toprovide an energy map; determining the energy of the number of halftonetile seams according to the energy map so as to indicate a number of lowenergy determined halftone tile seams, the number at least sufficient toachieve the desired resizing factor for the digital halftone image;performing a resizing of the halftone image by manipulating the numberof low energy determined halftone tile seams, to obtain a resizedhalftone image; and, printing the resized halftone image on a printer.33. The method of claim 32 where the methodology is only applied to asub-region of the digital halftone image.
 34. The method of claim 32where the number of low energy determined halftone tile seams arehalftone tile seams that possess energy below an energy threshold. 35.The method of claim 34 where the energy threshold is 5% greater than thelowest energy determined for any of the number of halftone tile seams.36. The method of claim 34 wherein the manipulating of at least one lowenergy determined halftone tile seam is a replication operation forincreasing the size of the digital halftone image.
 37. The method ofclaim 34 wherein the manipulating of at least one low energy determinedhalftone tile seam is a deletion operation for decreasing the size ofthe digital halftone image.
 38. The method of claim 34, whereindetermining the energy of halftone tiles is performed by summing thenumber of pixels within a tile that are of a different state thancorresponding pixels within its neighboring tiles.
 39. The method ofclaim 34 where the energy metric is based on the gradient of luminance.40. The method of claim 34, wherein determining the energy of the numberof halftone tile seams according to an energy metric is one of employingan exhaustive search and employing a dynamic programming method.